There is a high interest in accelerating multiscale models using data-driven techniques. Creating a large training dataset encompassing all relevant load scenarios is essential for a good surrogate, yet the computational cost of producing this data quickly becomes a limiting factor. Commonly, a pre-trained surrogate is used throughout the complete computational domain. We introduce an alternative adaptive mixture approach that uses a fast surrogate model as constitutive model when possible, but resorts back to the true high-fidelity model when necessary. The surrogate is thus not required to be accurate for every possible load condition, enabling a significant reduction in the data collection time. We achieve this by creating phases in the computational domain corresponding to the different models. These phases evolve using a phase-field driven by the surrogate uncertainty. When the surrogate uncertainty becomes large, the phase-field will thus cause the constitutive model to locally switch away from using the surrogate. If the surrogate uncertainty later decreases for that point, this approach allows a switch back to using the surrogate. We discuss the requirements of this approach to achieve accurate and stable results and compare the phase-field to a purely local approach that does not enforce the spatial smoothness of the phases. Using a Gaussian Process surrogate for an elasto-plastic material, we demonstrate the potential of this mixture of models to accelerate multiscale simulations.